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Preliminary information about a model system by which- state vocational 
education agencies may conduct p^ibdc program evaluations is presented. Another 
facet of the system is the facilitation of programmatic decision making through the 
incorporation of information about national and slate interests, student benefits, and 
manpower requirements. The discussion indudes a general system for self-xvriated 
and continuous program evaluation, vocational education objectives. program goals 
and data needed for measuring goal achievement, equation procedures and 
materials, and field tests of the system as conducted in the vocational educational 
agencies of three states (Colorado, Kentucky and New Jersey) which also cooperated 
in developing the model system. The revised system is undergoing further field tests. 
December 1969. is the projected project completion date. The model system is 
illustrated, and examples of information forms are presented. <CH) 



The Center for Vocational and Technical Education has 
been established as an independent unit on The Ohio State 
University campus vith a grant from the Division of 
Comprehensive and Vocational Education Hescarch, U. S. 
Office of Education- It serves a catalytic role in 
establishing consortia to focus on relevant problems in 
vocational and technical education. The Center is 
comprehensive in its commitment and responsibility, 
multidisciplinary in its approach, and inter institutional 
in its program. 



The major objectives of The Center foilew: 

1. To provide continuing reappraisal of the 
role and function of vocational and tech- 
nical education in cur democratic society; 

2. To stimulate and strengthen state, regional, 
and national programs of applied research 
and development directed toward the solution 
of pressing problems in vocational and 
technical education; 

3. To encourage the development of research to 
improve vocational and technical education 
in institutions of higher education and 
other appropriate settings; 

4. To conduct research studies directed toward 
the development of new knowledge and new 
applications of existing knowledge in 
vocational and technical education; 

5. To upgrade vocational education leadership 
(state supervisors, teacher educators, 
research specialists, and others) through 
an advanced study and inservice education 
program; 

6. To provide a national information retrieval, 
storage, and dissemination system for 
vocational and technical education linked 
with the Educational Resources Information 
Center located in the II. S. Office of 
Education. 
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PREFACE 



As vccaticna I anxi technical education has assumed rev prc~inar.cc and 
visibility, increased emphasis is being placed on acceptability through 
evaluation systems- State divisions of vocational and techni cal education 
are currently being ashed to implement state-side evaluation systems. 

These systems are essential for state- level program planning and for meeting 
Federal and state acceptability requirements - A state evaluation svstem 
model has been developed by The Center to assist state divisions in meeting 
these needs. The model has been designed to be sufficiently flexible to 
meet individual state needs, yet generate comparative data." This interim 
report is offered before procedures and ins trumention have been finalised, 
so the system and its components may be available at the earliest possible 
time for consideration by state staffs mho are initiating state evaluation. 

Ke appreciate the active cooperation and inputs of the Colorado and 
Kentucky State Divisions of Vocational Education *.*ho served as pilot states 
during the initial stages of the system's development. Acknowledgement is 
due Albert £* Jochen, formerly state director of vocational education in 
Mss* Jersey, and members of the advisory committee appointed by the National 
Association of State Directors of Vocational Education for their valuable 
insights and contributions to the system's design- recognition is due 
Richard Die f fonder fer , project associate; B. 3- Archer and Hark Fulccmer, 
research associates; O- J. Byrrside, ssbo served both as a consultant and 
research associate, and other Center staff for their -work on this project. 

Ue actively solicit the suggestions and comments of states, based on 
their experiences, directed toward the future improvement and refinement 
of the system. 



Director 

The Center for vocational 
and Technical Education 
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The central purpose of this project is to develop an evaluation system 
by which state vocational education agencies nay assess oaricdicaliy the 
effectiveness of their program efforts- The system also is designed to 
develco the information required for prcgrar 



itic cecisron na/.»nc wi in cue 



regard for national and state interests/ student benefits/ and manpower 
requirements - 



The project team included Center staff and an evaluation specialist from 
each of three cooperating states. The Center project staff Lad primary 
responsibility for system development- State evaluation specialists assisted 
Centex* project staff* in system development and were responsible for 
coordinating field tests of the system's components in their respective 
states- Several advisory committees , individuals representing many disciplines 
both within and outside of education/ and specialists from The Center were 
involved repeatedly in the project and provided guidance and assistance. 



The project staff first designed a general system for self-initiated, 
continuous evaluation of ststt programs- Then, a set or vocational education 
program objectives was formulated with the consultation and assistance of 
state directors and other professionals. State vocational education progr — 
goals required bv the objectives were developed and the data needed to measure* 
achievement of each program goal were identified- Procedures and materials 
needed by state staffs to effect the evaluation were prepared and a tryout 
of the system was conducted in the vocational education agencies cf the three 
cooperating states. 

Results from the field tests were used to revise program goals and 
data-coliectlon instruments- Hew instruments were devised to collect data 
from local schools and from other agencies, and the development of data 
processing procedures was begun. 



The revised system is being field tested and the final evaluation 
system will be completed in December of 1969. The system then will be made 
available to a wide audience of potential users- 



H SYSTEM FOR STATS £VALU7iTIOI3 OF VOCATIONAL EDUCATION 
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This report provides preliminary information about a model system under 
development biat will enable states to plan, monitor, evaluate and redirect 
their programs in vocational education- This intensi model Is offered berore 
completion of the major project from which the model is derived (The 
Development of a Kodel to Evaluate Scstc Programs of Vocational Education) 
so the methods and instruments developed in the course of the project may 
be available in time for consideration by s rate stafrs who must prepare 
detailed plans and evaluation systems in response to federal requirements . 
Additional tryouts, analyses and revisions are scheduled for the system 
described here, but the results which are presented provide the basis for 
effective evaluation using principles, procedures and instruments which 
have survived preliminary , but rigorous , trial - 

The first section of the report identifies najor operational 
characteristics of the system, explains the rejection of two evaluation 
methodologies in common use, summarizes important points of system development 
strategy, and gives an overview of project operating methods and arrangements - 
Subseauent sections describe the design of the system, its mitral field 
testing, its revision as a result of testing, and the development of machine 
<3ata processing procedures. An illustration of the use of the data system 
for program evaluation also is given. 



OPERATIONAL CHARACTERISTICS OF THE DESIRED SYSTEM 

The counsel of potential users of the model system and appropriate 
others was that an optimum system wc-uld satisfy state operating needs and 
be designed for use in the administrative mainstream as a management tool. 
That is the intended role for this system. It is designed to provide 
information essential for planning and for redirecting the programmatic 
efforts of state vocational education agencies- It will provide decision 
makers with information essential to the development of annual and long- 
range program plans. The evaluation system also is designed tp provide 
instruments and procedures for evaluating program effectiveness in ways 
responsive to the accountability requirements set by state vocational 
education governing boards, state legislatures, and the public. In 
developing the evaluation system, consideration was given to tne importance 
of periodic evaluation of vocational education called for at the national 
level by the 1968 Amendments to the Vocational Education Act of 1963, and 
to the reporting requirements of the U. S. Office of Education. Thus, a 
state evaluation system is being developed to collect, interpret and make 
readily available a body of information which is useful to a variety of 
agencies and to the public. 



EVALUATI CIS KETJiDDS Hi COI-IM&TJ USE 



The evaluation system described in this report was developed after 
careful considerat j on of methods in common use revealed that none of the 
existing met hods could supply data with the characteristics judged essential 
for effective program evaluation in the current operating situation of the 
states- Two of the major methods considered and rejected were: 

Traditional process evaluation . The evaluation methodology which has 
been used most by state agencies is a type of process evaluation where 
organizational sturctures, educational processes , equipment, and facilities 
are judged against pre-set standards. This methodology does not refer to 
program outcomes. Assessments of program adequacy are" determined through 
judgmental procedures and the opinion ratings of experts. This type of" 
evaluation has been used extensively for many purposes including the 
allocation of state resources to local programs and the development of 
budget requests to the legislatures. 

Although it is clear that the evaluation of program efficiency is <sn 
essential part of the assessment of a state program, it also is true that 
the value and efficiency of a program can be determined only with respect 
to an outcome. Thus, a program which is efficient for objective A may be 
inefficient for objective B, A facility, a teaching staff, or a budget is 
appropriate, adequate, or efficient only with respect to some goal; 
consequently, evaluations of the processes of education are important, but 
they are meaningful only after it is known whether the programs served 
their intended purposes. 

This point is being emphasized strongly in the political and legislative 
arenas. Hot only is the Federal legislation written to emphasize educational 
outcomes for specified groups of students, but education now finds itself 
having to compete at all governmental levels with other agencies and insti- 
tutions for limited human and economic resources. Decisions by policy- 
making bodies regarding resource allocations are being made with" increasing 
frequency on evidence of program effectiveness, relevance to social and 
economic conditions, and the degree to which programs reflect community, 
state, and federal interests and concerns. In such an environment, the type 
of process evaluation which has been traditional in vocational education 
fails to provide the evidence required by policy -mailing bodies and must be 
replaced by a more effective evaluation methodology if a proper case is to 
be made for support. 

The type of process evaluation which has been traditional also has 
certain practical and methodological difficulties. These methods require 
extensive local school visitations in order to secure evaluation data about 
programs, equipment and facilities. Such visitations require large numbers 
of manhours and personnel so that it is both difficult and expensive for 
any state to conduct frequent evaluations. As conducted, many such evalua- 
tions include insufficient attention to reasonable standards for validity 
and reliability of their instruments, to inter-evaluator reliability, and 
to sampling procedures. 

All-inclusive data banks . Another approach to evaluation of vocational 
programs has enjoyed popularity during the past decade. Ir. this approach, 
data are collected and stored which appear to have logical relevance to 
decision-making requirements. No systematic attempt is made to define data 
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needs in advance by reference to specific program planning or accountability 
ronni rege nt s Instead, data are accumulated in great variety vith tnc 
eSectatio* thatany data needed later probably will be available in the 

bank* 

^his aooroach to providing decision-maxing information to program 
manners caICbe markedly inefficient. Hot only docs it require Collec^c.n, 
processing and storage of data which, in large part, is not used, the 
data bank very likely will require frequent additions to meet n«sw, unantici- 
pated questions- The costs, delays in planning operations, disruption^©- 
normal activities at data sources, complication of data processing rowincs, 
and uncertainty of results which can result from frequent auditions or dhanges 
in data requirements suggest that a sabre efficient methodology should be 
employed if at all possible. 

SYSTEM DESIGN STRATEGY 

an evaluation nethodology capable of delivering the operational 

characteristics considered essential in the current situation of state 

vocational education planners apparently oust begin with a *?* 7 *°* 
to progran planning. This would require that: 1) the evaluation problems 

be defined in terns of the purposes and expected outcomes of prograns; 21 
an information (measurement) system be formulated to provide the particular 
data required for evaluative decisions about program outcomes; 3} feedback 
mechanisms be provided to permit monitoring of the effectiveness^ ana 
efficiency of the information system in providing s J 9 nixicant cata for 
decision making; 4) an interpretive system be formulated by which inioraatxon 
is analyzed and presented to decision makers in a format which facilitates 
decision making; and 5) since the evaluation system is only one partofa 
total program-planning system, the evaluation system be capable of ar-icula 
tion with other components (PPBS, for example) of a larger program-planning 

system. 



PROJECT ARRANGEMENTS AND OPERATING J-STHODS 

A hiqh level of involvement was solicited from various agencies, groups 
and individuals, most cf whom were external to The Center. This was done to 
assure the development of realistic evaluative instruments, to avoid 
duplication of efforts, and to maximize yield through an interdisciplinary 

approach tc the total problem. 

Cooperative agreements were established between The Center and the 
states of Colorado, Kentucky and New Jersey whereby each director of 
vocational education identified a full-time evaluation specialist wno was 
iointly employed by the state and The Center. These specialists were 
available in their states and for workshops at The Center to assist m the 
development of the conceptual framework, the instrumentation and procedures, 
and to act as liaison with state staffs during field-testing within their 
spates Their participation in the project helped assure that the 
developing model reflected realistic, immediate and long-term evaluation 
requirements of the states* 

From May of 1967 through June of 1968, regular workshops were scheduled 
at The Center for state evaluation specialists to meet with The Center 
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project staff. At these workshops, tlse state evaluation specialists 
reviewed the materials being developed at She Center, they assisted with 
the continuous process of reformulating and refining the conceptual scheme, 
system elements, and the evaluatien instruments and procedures. 
S i n ? s She Center project staff has been devoted to completion 

of developmental activities required to finalize the system. Evaluation 
socialists have continued to react to developed products and to coordinate 
field-test activities within their respective states. 



Three aavisory committees reviewed project plans, evaluation materials 
and procedures, and suggested alternative conceptual approaches. 1 These 
committees were: an advisory committee of state directors of vocational 

education, selected by the National Association of State directors; a 
committee consisting of recognized authorities in the vocational education 
profession with expertise in such areas as administration, supervision, 
teacher education and research; and a third cosiamittee c©iig>©sed of Center 
specialists. The members of the advisory committees are listed in Appendix 



Inputs also were sought from other expert sources excemel to The 
Center* including the staff of the United States Office of Education and 
j~ e staff of other vocational education research and develpf&oent centers^ 

The report and recommendations of the National Advisor* Council cn Vocational 
Education were reviewed and the evaluation model's data system was des igned 
to include information needed from the states for national evaluation of 
'ocational education. 
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Earing ac first phase cf the project (JCr; B67 to SCrcster If £33 , 
major project activities were ; 

1. conceptualization cf a model system; 

2. formulation cf objectives for state vocational education programs 
and the identification of quantitative program goal statements 
«hich are used to assess achievement of the program objectives; 

3. specification of data needed to measure the extent to which program 
goals are realised; 

A. development of procedures for using evaluation results to develop 
annual and long-range program plans; and 

5- field testing of the system in the cooperating states. 



COKttfTm SGSEME DEVELOPMENT 

5he first step in conceptualizing the evaluation system was to define 
the purposes for which the system would he used by state program planners - 
Three major uses were identified. The system would be used to provide 
information of assistance to states in; 13 redirecting programmatic efforts 
2) planning annual and long— range programs# and 3} meeting accountability 
requirements - 

The next step was to organ! ze a system to accomplish these purposes 
within a state vocational education setting. Review of existing method- 
ological approaches to program evaluation convinced the project staff that, 
in order to serve the above purposes, an evaluation system would have to be 
developed which would be compatible with a systems approach to program 
planning. Thus, the project staff undertook the formulation of a conceptual 
scheme which would contain problem-defining elements such as objectives and 
goals, problem-solving elements including the means by which alternative 
strategies could be evolved for accomplishing goals, and an evaluation 
methodology required to measure the effectiveness of efforts to achieve 
specific program goals. 



USING EVALUATION RESULTS FOR PROGRAM PLANNING 

Fig ur e 1 shews the sequence of major steps required in the evaluation- 
pregram planning cycle- A state using" the model will have available a set 
of program objectives and goal statements {step 1} , a data system with 
computer programs and procedures (step 2 } , and evaluation instruments and 
data flow procedures (step 33 . These first three steps have been developed 
by The Center project staff- The state would review the program objectives 
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mud goal statements for the purpose of selecting those it found useful for 
its needs. la cs£&g the model for the first tine, the state has the option 
of assigning specific goals in quantitative terms to each goal statement 
Istep 4f or cot assigning specific goals and using evaluation results to 
establish a base line Cor future goal projections. 

Evaluative information would then he collected (step 5| and the data 
analysed in terns of goal accomplishments (step $). Seasons for discrepancies 
between projected and actual goal accomplishments would then be assessed 
i. failure to initiate planned strategies, cnforseen obstacles 
(stc-j 7J . Steps 5, t, and 7 male up the specific evaluation phase of the 
total system- 



program planning activities begin with the redefining and redevelopment 
of new information requirements (step *! . This is followed by a reformulation 
of new data requirements, if necessary (step 51 . 1 series of additional 

program planning activities (steps 10-141 are then carried out by appropriate 
state staff culminating in the selection of strategies (step 15 J designed to 
accomplish the newly formulated annual and long-range goals of the state 
vocational agency. 

following these program planning activities the various selected 
strategies for achieving the new goals are implemented (step 15! . After a 
time span, evaluation procedures (steps 5, fi and 7J are again carried out 
and the program planning procedures are recycled. 

This system and its procedures respond to three requisites. First, 
it was required that the evaluation system be designed for self-initiated 
evaluat^n which would contribute to decision making involved in state-level 
program planning. Second, the system was to relate program outcomes and 
specific program goals as a logical basis for planning and replanning 
activities and programs, third, the evaluation system was required to be 
a continually operative mechanism in order that program plans might be 
adjusted whenever required by changes in the field situation. 



FtfRKULATIOfi OF PROGRAM OBJECTIVES A HD GOALS 

the evaluation plan requires first that a set of objectives be 
formulated which define the programmatic thrusts of the state agency. ‘Then, 
for each objective, a set of specific, quantitative goals must be stated, 
such that achievement of the goals constitutes achievement of the objective. 
Program objectives and goals must be formulated sc as to support the program 
decision requirements of the state agency. 

‘The evaluation model allows for the development of program objectives, 
goals and data requirements independently by each state. However, in order 
to test the usefulness of the evaluation system and to provide states with 
a workable starting point, a complete set of program objectives, goal 
statements and data requirements were developed by the project. Objectives 
and goal statements are listed in Appendix B and samples of the forms for 
data collection are shown in Appendix C. 

As program objectives were developed, they were reviewed periodically 
by the advisory committee of Center specialists, the external advisory 
committees, and the state evaluation specialists. Recommendations of these 
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groups verc influential is ibsassa&Aftl&g sis preliminary program tbjectlces 
which then were sent CSettber 1967? fer rcvi.cs to ail state directors of 
vocational education and to bead state supervisors of vocational educati on 
la Janaary of 15£2, the returns from this national survey sere compiled 
and a final set of fear program objectives was produced shies can he 
accepted hy many stales - 



A set of goal statements then was formulated for each program objective. 
These quantitative statements sere designed to permit measurement, of the 
extent to which = a) target populations of concern to vocational educators 
are being served; bj local schools assure program quality; c3 programs are 
accessible to students; and d) state vocational education agencies use 
student characteristic and follow-up data in their planning. For illustra- 
tion, consider objective three appendix &} , “To provide increased 
accessibility to programs of vocational education to meet the needs of those 
to be served. “ As stated, the objective is a summary of many specific goals. 
If ve want to decide whether the objective has been met, we must examine 
numerous specific facts, she gc-rl statements listed for objective three are 
the specific facts judged essential to a decision about achievement of the 
objective. Thus, amung other facts, it is important to know (Coal D) how 
many students failed to complete a vocational program because of transpor- 
tation difficulty or because they could not meet instructional and related 
costs (Coal £) - Once a state has determined its starting situation with 
respect to each goal related to an objective, it can set targets for the 
next period and assess improvements objectively. 



FQPJS3lATi€Si Of SATA ?i^3K£M£!JTS 

In this model, data requirements were derived systematically from 
evaluation purposes, program objectives and program goals. In developing 
the data set, careful attention was directed toward selecting the minimum 
number of data elements which would assess the extent to which program 
goals were accomplished. The program goal statements required the formula- 
tion and organization of a data set which would provide quantitative 
indicators of target population needs and training outcomes, school and 
community characteristics, and vocational education programs and processes. 

State-level program planners require evaluation results to plan and 
redirect agency activities and to selectively allocate personnel and the 
financial resources required to accomplish these activities- Therefore, 
the data set was organized in ways which would have the greatest usefulness 
for these management operations. To accocplish these aims, the data system 
was organized to provide information which could be analyzed by program 
sectors (public, private), program levels, program areas and where applicable 
by facility types, counties, educational planning districts, student ar*n and 
grade level. The term “educational planning district" as used here refers 
to regions within a st3.ti having common economic, social and population 
characteristics such that they warrant separate attention in program plarning 
A state using this evaluation system would determine its own educational 
planning districts. 

As the Odta set evolved, it was recognized that data inputs to the 
system would have to be obtained from a variety of sources. These sources 
would include schools, students, and state and local non-school agencies. 

As a result of the potential complexities involved in obtaining data, three 
additional efficiency factors received attention during the process of 



foraclitisg th« data set. 'Shese factors were £he time £ raise recuired f cr 
c^Llectaag cata, the costs to the state agency cf catling cut data collection 
procecures , and personnel required to manage the data collections - 



FUEL© TEST ?£C££©1*RES 

^ c f* 1C5 ? ?tfa * i *«^*me was developed and the project staff had 
foredated a preliminary data set, field tests of the -Sfcerials were conduct- ed 

f 1 *? 6 * in 42,8 Spring of 196$. Center orcjcct staff visited 

fZT‘ “* v- “ 

1. evaluate the relevance of the formulated program objectives, 
program goa^s and data set for state program^planning purposes: 

2. identify the states* procedures for obtaining and reporting data 

required by the system; and ^ 9 

3. conduct a simulated test of the program planning procedures- 

^ Shese initial field-test activities were carried out within the state 
™ ^ in J COOperatl ° n With othe * * tate-level governmental aoencies It wL 

** “VOlve local school* in the evaLa£f^ “occdur« 

. 4lBe * , Fxeld testing at the state level was c ons idered to he necessary 

before the evaluation system was expanded and tested state-wide. ^ 

FIELD TEST RESULTS 

in u, r^“ ° f s y s ! g *‘_ 00ai>0 ' >en n fcr ? roqraffi pianino. Professional staff 

«*?**» critically evaluated tke program objectives, goal 
an d th e ^* ta set for relevance to their own program-plannina needs 
.nose system components went through a series of xevisio nsbv the rroilct 

so ^ y J^ C state staffs during the field' tests!? Even 

a of * ddlt ^ 0 ’' a l changes in the program goal statements and the 

have at the end of field test before the system would 

have maximax effectiveness for program planning. 

of u r^T fi lo>t P j OCgdu f gs rvquired for t he systen . A study was made 
an i i* to Which data already on file in the state vocational agencies 

svs tin • ^ 'H*^ S ^ atC ~ leVel agencies °° uld 'ised to meet the evaluation 
S Stances, data collected and organized 

used effief IS™*!? state vocational reporting requirements could not be 
M . efficiently or reorganized to meet the requirements of the data system 
Much of the information about target populations which was requested frl^ 
er state ajrncits was unavailable. In addition, nany of these state-level 

existino ^ mU * mK J . i **** ***** greyer lead-^ tf orgar^ze 

i^teiSs re ? u *? ted b y **« state vocational agency, or, in some 

i ( tances , that the data simply were unavailable. 3 

wool /hilt S e jf _ ^ est , e3¥ ; rie 5“* “ ade it clear that an information system 

elsefttel data b f^ V ?"° I T 1 3 C d J hat new instruments were required to Collect 

te b^ artiCTtetfd^iewV 1 *? 01 * ources - ®»« ne “ instruments would need 
to federal and state reporting requirements, designed 

essential for reliability of^asurement* ^ 3 ™ 6 ' " eet technxeal standards 
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Siniatisn tht program planning proce gores. As noted abo^e, scuff iciest 
evaluation data conic sot be collgctcc or or^iaiigd iron existing sources to 
permit a realistic field-test of progrzaa-planning procedures. However, state 
staffs is the cooperating states provided critical evaluations of the 
suggested approach to pa»gra*c-piasaing procedures. She reactions obtained 
were considered favorable. 

Additional benefits frees field-testing. Project staff involvement with 
state supervisors provided many insights useful for increasing the efficiency 
of the model. Also, state personnel became more aware of implications and 
benefits of the evaluation system for their service areas or unit operations - 
Channels of communication were either established or broadened between the 
program— planning and evaluation personnel in the state vocational education 
agency and their counterparts in other state agencies who were as Seed to 
supply data for the evaluation. 
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PHASE II: REVISION OF TIE B81LUATIGM SYSTOl 



Field testing of t he evaluation model provided the reality check 
needed for inproving the model's effectiveness and efficiency,. After the 
results of the field-tests were assessed, the following objectives were set 
for the second and final phase cf aodel development: 

1. a data system compatible with state and national needs; 

2. data processing programs and procedures for use with the data 
svstem; and 

3. field testing and validation of the total model in a number of 
participating states to assure that the final system is effective 
and efficient in terms of its purposes. 



FORMULATING THE REVISED DATA SYSTEM 

Program goal statements and the existing data set were reexamined and 
revised in light of field test results, evaluation needs called for in the 
1968 amendments to the National Vocational Education Act, and U. S. Office 
of Education requests for state evaluation data. The resulting data set 
has been incorporated in seven data-collection instruments. Appendix C 
displays the face sheet of each of the six instruments used to collect data 
from locr.l sources. Four of the six instruments are designed for completion 
by local school administrators and vocational teachers. Together, these 
four instruments (DF-1, DF-2, DF-3, DF-6) provide information about vocational 
program characteristics, schools and communities. Two CDF-4, DF-5) of the 
six instruments provide data about individual students and former students. 

The six instruments are described below. 

A Vocational Program Status Form (DF-1) is designed to identify the 
location of vocational education programs within the state. Information is 
collected about reimbursed and non-reirabursed vocational programs. This 
form is completed once at the beginning of each school year by a school 
administrator or his designee. Private trade and business schools are 
requested to respond to selected items from this data form. 

A School Information Inventory (DF-2) is designed to secure information 
about school and community characteristics and local vocational education 
program planning activities. It is completed once at the beginning of each 
school year by a school administrator or his designee. 

A Specific Program Information Data Form for Secondary and Post- 
Secondary Instructional Programs (DF-3) is designed to secure vocational 
education program information for all vocational programs at the secondary 
and post-secondary levels. This form is conpleted once at the beginning 
of each school year for every instructional program which is offered. 
Additional DF-3 forms are forwarded to the state office during a given school 
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year whenever: X) a new instructional program is started or 2) a new section 

of an already existing instructional program is offered at a new level (e.g. , 
an existing instructional program at the secondary level is now also offered 
at the post-secondary level) . The DF-3 is coa^leted by an administrator or 
a vocational teacher. 

A Specific Program Information Data Form for Adult Instructional 
Programs (DF-6) is designed to secure information about adult instructional 
program characteristics. This form is completed once a year for each adult- 
level instructional program which is offered. Additional DF-6*s ate 
forwarded to the state office whenever a new instructional program is 
started or whenever an existing adult instructional program is recycled. 

This data form is completed by a vocational administrator or vocational 
teacher. 



A data form in two parts is used to secure student characteristic data, 
provide continuous monitoring of student status, and serve as a basis for 
selecting student samples for follow-up studies. 

The two parts of the form are labeled Ctudent Information Card (DF-4A) 
and Vocational Student Status Report (BF-4E) , All students enrolled in 
vocational programs and having a vocational objective are expected to 
complete form DF-4A. Completed forms are sent the state agency. For 
each returned DF— 4A, a DF-4B is prepared which contains printed information 
to identify the student, school and program. These forms are returned to 
the instructor responsible for the vocational program. It is then his 
responsibility to return the DF-4B to the state agency either at the end of 
the school year or whenever there is a change in the enrollment status for 
any student. In states having very large vocational enrollments, the use 
of the Vocational Student Status Report can be handled on a sampling basis. 

Experience of states using similar forms indicates that students can 
be expected to complete a DF-4A in 20 minutes or less. During the first 
yoar of operation, additional class time will be required to orient students 
to the purposes of the form and to assist students in obtaining social 
security numbers . 

The Follow-up Survey of Former Vocational Students (DF-5) is designed 
to secure information from persons who completed their vocational instruction 
or terminated a program prior to completing program requirements. The 
student follow-up procedures are initiated by the state agency rather than 
by local school districts. The model permits the state agency to conduct 
comprehensive , long-term, follow-up studies of program terminees for state 
program-planning purposes . 

A seventh instrument. State Division of Vocational Education Evaluation 
and Planning Data (DF-7) , is designed to provide information principally 
about target population characteristics and training needs. The data called 
for in this instrument are compiled by state agency program planning and 
evaluation personnel. The data are secured from other state agencies. 

The instruments are being pilot tested to identify any semantic 
problems in the items and to determine the time required by school personnel 
to complete the various data forms. Those local personnel who have been 
involved in reacting to the revised instruments to date have indicated that, 
in most instances, they find instrument length and the time required for 
their completion to be quite reasonable. Their specific reactions were 
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used by the project staff to oiniai2c the lumber of items and to revise 
wording where necessary. 

In developing the seven data collection Instruments, careful attention 
aas paid to the information flow requirements of the system. Figure 2 
depicts this information flow network. It mil be noted that evaluative 
data flows from local school sources to the state vocational agency and 
back again. 

The project staff be-ieves quite strongly that increased data 
reliability can foe obtained by providing local personnel with a feedback 
of evaluation results of interest and concern to them. Therefore, local 
school administrators who have reviewed the instruments were asked to 
indicate the types of feedback that they would consider important for their 
planning needs and provision has been made for appropriate reporting to 
them. 



MACHINE PROCESSING PROCEDURES 

The data collection instruments are designed for machine processing 
and data analyses. The use of machine processing will permit analyses cf 
the relationships between students* characteristics, program processes, 
and outcome indices. It also will permit rapid retrieval of information 
for use by program planners and various breakdowns of data which would be 
c-f concern tc state and local level program planners. 

The present organization of data calls for all program and student 
information required by the data system be collected at the local school 
level. These data then are summed by program and program area for all 
schools within a county. County tabulations are summed into educational 
planning district totals. Educational planning district totals are summed 
in turn to provide the total state data picture. 

Printouts of appropriate data will be provided to local schools 
involved in the state evaluation effort. These printouts will contain 
school totals, county totals, and education planning district totals for 
the county and district within which each school is located. 



ILLUSTRATING THE USE OF THE DATA SYSTEM 

Program Objective One, goal statements M and N (Figure 3) are concerned 
with the numbers of economically disadvantaged persons who are enrolled in 
vocational programs. 

Having set specific gcals for th statements, the state agency wishes 
to know if it has achieved its aims. * ormation to answer the question is 
obtained from the use of the Student Information Card (Figure 4) items 
12 and 13. 
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By tabulating the data in the Banner already described, the vocational 
division can determine the extent to which it has achieved its present goal 
of providing vocational education t© these specific target populations. 

Other inforaation obtained from the evaluation instruments will permit 
the state agency to determine whether training took place in regional areas 
of concern, e.g. , specific educational planning districts, identified 



economically depressed areas. The stepn outlined in Figure 1, the 
Eviluitioo'Proqrai Pinning Cycle would then be initiated; net# annu a l 
and lcog tern program goals developed; strategies for their attainment 
implemented; and the evaluation process recycled. 

Suppose, as an outcome of training, the state agency is concerned vith 
the effectiveness Of vocational education in increasing the economic well- 
being of economically disadvantaged persons; economically disadvantaged in 
the %mnsc of either being welfare reoeipients or having been previously 
employed at a wage rate below the federal minim u m wage standard. Our next 
step would be to conduct a follow-up study of former students identified as 
coming from these target populations. This would be accomp lis hed through 
the use of the Student Information Card described above. The Vocational 
Student Status Beport (Figure 5) permits breaking-down the total sample of 
these identified former students into subgroups by program including those 
who comleted their t raining (item 1) , those who terminated their instruction 
prior to completing program requirements (item 1) , those who participated in 
out-of-class vocational experiences (item b) , and those who received 
specialized remedial instruction during their vocational training (item cj. 
Further, it can be determined if these economically disadvantaged pe rson s^ 
ara also ifrfifri of Minority groups* This inforsitlon is githertd from the 
Student Information Card (item 10) . 
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FIGURE 5 CCOJJT . j 
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The target populations would receive a Follow-up Survey of Foraer 
Vocational Students (DF-5) - Data about initial post-training earnings, 
as well as current earnings of those sanpled, would be secured using itens 
7 and 10 (Figure 6 ) . 



FIGURE 6 

OF- 5 A FOLLOW-UP SURVEY OF FORMER VOCATIONAL STUDENTS 
/ ITEM 7 

7. If you cot a Mi-tie* job when you left this vocational 
program: 

a. What was your job? 



b. What did you do? 



c- What was your hegi—ieg h eart y wage on your first 
ft» M tine job since leaving this vocational program? 

~ ] Never had a full-time job 

§ J $L59 or less per hour 
□ $160 to $1.99 per hour 
"j $2.0) to $2.49 per hour 
$2.50 to $2.99 per hour 
$100 to $3.99 per hour 
$4.00 or more per hour 
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FIGURE 6 (CGJ2T.) 

OF- 5 A FOLLOtf-W* SURVEY OF FORMER VOCATIONAL STU0HHT5 

ITEM 10 



10. If you are present*/ wofSrihg: 
a. WJjjI is your job ? . 



b. What do you do? 



c. What wages are you present!/ earning? 




Not working atihe present time 
?L59 or less per :jout 
$L50 to$L99 par hour 
$2-05 to J2.49 per hour 
J2-o0 to $2.99 per hour 
$105 to 53.99 per hour 
$L0G or more per hour 



Other data from forms DF-3, DF-4A, DF-6, DF-4B, and DF-5 also permit 
analyses of relationships between specific programs, other student 
characteristics and economic outcomes. The data system and the manner in 
which data are organised in terms of local to state summations permit a 
determination of which vocational programs, by regions, were most success- 
ful in providing the target populations with skills which led to higher wacie 
earnings. 

This illustration shows how the data system is used to measure specific 
goal outcomes and how it provides a base for more specialized studies to 
measure program effectiveness. The main point which should not be over- 
looked is the systematic manner in which the state agency proceeds in the 
evaluation cycle; that is, it starts with objectives,, derives goal statements 
which explain how objectives are intended to be achieved, assigns specific 
goals (outcomes) to be achieved, measures the degree of success in attaining 
goals, analyzes failures in goal attainment, and systematically initiates 
program planning procedures to redirect state agency efforts in achieving 
new annual and long-range goals . 



FIELD TESTING AND VALIDATION OF THE SYSTEM 

Present project plans call for field testing the revised instruments 
and procedures in cooperating states beginning in July of 1969. Selected 
educational planning districts within each of the cooperating states will 
be used for the initial field tests. Following a review of the initial 
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findings, a final set of instruments and procedures will be produced by 
Sep temb er of D69. Tne cooperating states then will conduct a state-wide 
test of the system's instrumentation and information flow procedures. 

Data obtained* from the states will be transmitted to The Center for data 
processing and analyses. A limited test of the program planning features 
of the model then will be initiated in each of the cooperating states - 

Following this? final testing of the instruments and procedures, in 
Decenber of 1969, a final evaluation model will be produced at The Center 
and the instruments and procedures for their use will be disseminated to 
potential users. 



25 



/iPPBiDICES 



o 

ERIC 









APPENDIX A 



AOVISCPJ COKKITTEES 



CONSULTANTS 

Alters £. J oaken , former Assistant Commissioner for Vocational Education, 

New Jersey 

i. Robert Varrbrod, Professor, Agricultural Education, The Ohio State 
University 

0. J. By r ns ids , Executive Secretary, National Business Education Association, 
Washington, D. C. 



PILOT STATE EVALUATION SPECIALISTS 

Joseph 2Z. Bailey, Associate Director of Vocational Education, Colorado 
Lm siergenreter, Assistant Supervisor, Research, Colorado 
L. C. McSowell , Teacher Trainer, Kentucky State Department of Education, 
Kentucky 

Charles Gibson, Director of Program Planning and Evaluation, Kentucky 
Charles Beal, Director of Program Evaluation, Kentucky 
Prank Pinkowski, Director of Program Planning, Hew Jersey 



PROJECT ADVISORY COMMITTEE: NATIONAL ASSOCIATION OF STATE DIRECTORS OF 

VOCATIONAL EDUCATION 

J. B. Cullison , State Director, Arizona 
Srnesi G. Kroner, State Director, Washington 
Joseph P. Murphy , State Director, Connecticut 
Byrl Shoemaker, State Director, Ohio 
Cecil £. Stanley, State Director, Nebraska 
Sam K. Vick, State Director, Minnesota 

B. D. Anderson, State Director, South Carolina 
Paul M. Hodgson, State Director, Delaware 
Billion G. loomis , State Director, Oregon 
Valter Markham, State Director, Massachusetts 

PROJECT ADVISORY COMMITTEE: PROFESSIONALS IN VOCATIONAL EDUCATION 

Alma Bentley, State Supervisor, Home Economics Education, South Carolina 
Sara Blackwell, Director, Home Economics Education, Cornell University, 

New York 

Harold Byram, Project Leader, Research and Development Program in 

Vocational and Technical Education, Michigan State University, Michigan 
Joseph B. Clary, Director, Research Coordinating Unit, North Carolina 
A.lecne Cross , Director, Home Economics Education, University of Georgia, 
Georgia 

C. Thomas Olivo , Director Division of Industrial Education, State Department 
of Education, New York 

Jerry Srhulsier, Superintendent, Division of Employment Security, Department 
of Labor and Industry, New Jersey 




ft *4 W <CJ 



PROJECT ADVISORY COKXITT EE: 



CEUTER SPECIALISTS 



£c 2 -i £• £*xxpzell. Occupational Psychology 
f. smarting Occupational Sociology 
kc, Hose Eccnonics 
al 7L*ijK, Distributive Education 



23 



APPENDIX 3 

PROGRAM OBJECTIVES AND GOAL STATEMENTS 

l»h 3 Cc.t i vc* Cce 



T© provide V0£dfi0£4l edoc-ist ion S© y©y § * and Sd«s ft at-.© ail 3 te efiferina 
the 3 eL©r end t© th©S© wh© seeifc f© upgrade their ©c© upa 1 i ©ns 3 

conpefesjcies ©r learn new *>§3!©. 



GOALS 


A 


# puhl * c Secondary school youth ere enrol ied in vom- 

fiens! programs. 


8 


S puh I i C secondary inhno i youth with nhvsi ra ! or saents S 

handicaps ere enrolled in vocations! progress. 


C 


S disadvantaged public secondary school vowtia are 

enrolled in vocations! progress. 


0 


% 

S persons a reenrolled in post-secondary preparatory 

vocational programs. 

* ... __ 


£ 


41 

* persons with physical or mental handicaps are 

enrolled in post-secondary preparatory vocational programs. 


F 


# dssadvantaqed persons ere enrolled in post-secondary 

preparatory vocational programs. 


G 


.f persons are enrolled in adult preparatory vocational 

programs. 


H 


§ persons are enrolled in adult supp I ementary voca- 

tionai programs. 


I 


§ persons with physical or mental handicaps are 

enrolled in regular adult preparatory vocational programs^ 
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5 * disadvantaged persons are enrolled in regular adult 

preparatory vocational protracts. 


r 


§ persons with physical or rental handicaps are 

enrolled in "special needs" adult vocational programs- 


L 


s disadvantaged persons are enrolled in adulx "special 

needs" vocational prog rasas. 


U 


§ persons whose hourly wages for full-time employment 

or (o^ the ffidfirol nin issues ^052 stsnd^rd 9^ Oiirol 15 ^ 

in voC3tion^l progriscjS to i^cr£<)S£ fh{t ir hourly g^rnifigS. 


?3 


£ persons receiving public assistance are enrolled 

in vocational programs. 



Ob % active Two 



To proving coiaprc'tiSiisivg curricula which re I axe general and vocational 
education offerings to the vocational objectives of students. 



GOALS 


h 


| vocational proa rasas have their vocational and 

general education staffs jointly organise their offerings in 
relation to vocational program object ives. 


8 


§ vocational programs have indicated when they next 

plan to analyse their curriculum to update the occupational 
competencies that need to be taught. 


C 


I vocational programs use written performance ohjec- 

tivos based on an analysis of required occupational competencies. 


0 


3 vocational programs provide students with consumer 

education os part of the total i nstruct j ono J program. 
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Sfejggjjvg Three 



To provide fpcrcfisc-i access i 5? « I i t v to p re 5 r ass of vocational educe* !.•«. 
meet *'he needs of those ?*> pa served . 



£:*£LS 


A 


# vocational programs enroll students with educational 

deficiencies which couid act as serious harriers to successful 
program completion or job placement. 


6 


$ vocational programs sake available specialised edvea- 

Lionel instruction to students with educational deficiencies 
which could act as serious barriers to successful p rogr.m 
completion or job placement. 


C 


j? student* who did not complete f?aeir instruction did 

so because of inability to meet inslructionai and related costs. 
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?' students who did not complete their instruction did 

50 because of unava i labi I i ty of transportation. 


<•* 
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8 new vocational programs approved to receive state 

funds are operational. 


F 


8 vocational programs enrol 1 the potential number ©f 

students they can accommodate. 




f public schools offer vocational programs during 

both day and evening hours. 


H 


8 vocational programs operate at times other than the 

regular school year * September to May or June). 


1 


8 public schools offer vocational group guidance. 
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8 public school students receive vocational group 

gu i dance. 
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S new vocational programs are planned to become 

operational in areas indenti fled as being economical iy depressed. 
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i psjibl 5c School s ?<$ve pstfvuel cr- 5 . * r g f3*?<r pl-a-fis ?>. r 

local vtiwJiecii edueas 5<.n pro^rgat-c. 
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M pu&3 ic school 2 offering v^o-sf ic££i pro gr 52 r& *-%v<* 

placement ssrv i ces $ v* i S $& § & So $11 program f*rmisie$s. 




# voc« f i on* i programs provide Coopfifl'SS'iv-sf wor* 

exper i ences So students. 


0 


i vocational pr©gr$ms provide 5 i«u listed w©r & 

expcrim«5 in the classroom setting* 


£ 


i? public schools ©1 fieri ns vocational programs iiavo a 

general advisory ©©sasafts&e for the v©ca t i © 0 $ 3 program®. 


F 


£ vocational pros race have an adv ; sorv ©easaS ttes. 


15 


£ vocational programs actively use an edv i -;srv 

50 jsisifV$g in sv4 Seating fhsir ©pars t ion?. 


H 


# vocational program comp lot ions who sought full-time 

ficpioystnt. found sucs> cap i ©ymer t in the occnpaf ion for which 
tii?y »er* trained. 


I 


f vocational program completions who sought full-time 

employment, found such employment in an occupation related to 
that for which they were trained. 


1 

-J 


£ vocations 1 program terainees who did not coop lets 

program requirements and who sought full-time employment, found 
such employment in the occupation for which they were trained 


X 


# vocational procram terainees who did not complete 

program requirements and who sought full-time employment, found 
such employment in an occupation related to that for which they 
were being trained. 
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J wca- iocaS program cc®p let lens having had acre toe® 

esfi foil-tin* jc^ caatiao* to be £ 12 ? toy -ft -5 5a the cccqpaf Sen tor 
which t'fcey were f rsir,«iS. 


S3 


Jr _ vocational program cccpleticos having had ascf« Shan 

ce* full-time job continue to be employed “a an occupation 
relaSed to thaS tor arts i ch they were trained. 


U| | 

maufe 1 




j? vocal ion a 5 programs 535 * amortisation jcnt^i«5 tor 

a 1 5 major ogu i pmen S . 


© 


3 psj & lie schoe i s wbiefe anv« vocational programs 0 ? f*r 

Special 5 2ed vocational guidance services 'S'© noncol 3ege-b©us*d 
StodemlS. 


3* 


3 5 tots cart 5 f guidance |j«f5cssa«l »svi special VOCS- 

lion a 3 2 «iv*a« preparation. 


$ 


3 vcca3 ional teachers have completed state certifi- 

©at lea requirements tor the i as tructi ©as! area in which they 
teach. 


ft 


3 vocational programs have i ad Seated when they next 
plan to secure manpower information as a means for estimating 
potential amp i oyment opportunities for program 3 trainees- 


— 

3 


5 vocational students identified as having educational 

deficiencies which could act as serious harriers to successful 
program completion or job p i element receive specialised educa- 
tional assistance. 


T 


1 vocational program completions were sufficiently 

satisfied with the training they received they would recommend 
the vocational program to others. 


13 


jj? vocational program completions have enrolled for 

additional vocational education to upgrade their previously 
learned vocational s*sii?S- 


V 


3 vocational program terminees who did not complete 
their program regu i resents have since enrolled in a vocational 
education program. 


I <w 


3 reimbursed vocational programs prepare persons for 

occupations paying more than the federal minimum wage. 
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APPENDIX C 



FACE SHEETS OF EVALUATION SYSTEM DATA FORMS 



Appendix C includes only the face sheets of the instruments used to 
oollcct dete from local schools . A revised version of these date forms is 
being used for the final field test of the system. 

Data Forms by Title: 

DF-1 Vocational Program Status Report 
OP-2 General School Information Inventory 

DF— 3 Specific Program Information Data for Secondary and Post-Secondary 

Instructional Programs 

DF-4A Student Information Card 

4B Vocational Student Status Report 

DF-5 A Follow-up Survey of Former Vocational Students 

DF-6 Specific Program Information Data for Adult Instructional Programs 




VOCATIONAL PROGRAM STATUS REPORT 
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DM GENERAL SCHOOL INFORMATION INVENTORY 



Fcr S5r!e Us* C’V 




Name cf Scfcci 1 



School District 



Background Information 

1. Dees the school curriculum require all students to 
receive seme exposure to marketable skill training? 

□ Yes QH o 

2. How many hours per day is the school in operation? 

Hours 

♦ 

3. How many hours per day are vocational programs f 

offered? Homs 

4- Does the school have a practical arts program (s)? f 

□ Yes j 

Check types of programs: 

□ Industrial arts I 

□ Genera! business 

□ Home economics (hememaking) 

□ Other 

(specify) j 



firm cn □ m 



Person Ccrrg’et'rg This Form _ 



P sitxn _ _ _ _ Date _ . 



8. Dees the school have a formally structured occupa- 
tional information program? 

□ Yes 

Specify types cf students served: 

□ Vocational students only 

□ All students 

□ Ho program offered. 

9. Does the school provide job placement services to 
vocational program ierminees? 

□ Yes 

Indicate type(s) of placement services available 
within the school: 

□ Cooperative program between school and 
state employment service 

□ Vocational teachers make referrals 

□ Guidance Counselors 

□ Other (specify) 



□ No placement services avaflable. 



□ No programs offered. 



Ancillary Services 

5. Does the school have a guidance program? 

□ Yes □«£> 



6. Does the school have a specialized vocational guid- 
ance program fcr noncoHege-bound students? 

□ Yes □ No 



7. Total number of guidance personnel. |~|~j 

a. How many are state certifed? j~r~j 

b. How many of these slate certified persons 
have special vocational guidance prepara- 



tion? 



m 



, o 

me 



10. Is prevocafiona! instruction/orientation provided to 
students in this school? 

QYes 

Note jype of program and type of enrollment: 

Type of Program 

□ As a specific course. 

□ Integrated into other course work. 

Type of Enrollment 

□ Elementary students only (grades 8 and be- 
below). 

□ Secondary students (grades 9 and above). 

□ Both elementary and secondary students. 
Total Enrollment 

□nf l j Students receive prevccational in- 
struction. 

□ No prevccational instruction /orientation provided. 




DF-3 



SPECIFIC PROGRAM INFORMATION DATA 
FCR SECONDARY AND POST-SECONDARY 
INSTRUCTIONAL PROGRAMS 

(Fen lime Training Program) 



rm m o ro 

f3t!£*w5e£*:> 

i T~i~ iTTl 

Vbc*1-.3R»j £z£e 



Date 



School Na s 3 Perscn Riling Cut This Rrim — - 

School Dlstric - Position 

NOTE. if this trainin', s nci . mduu^d h. a regular :oo! ^ tuaticn. ■ :s° the space above to llsi the special program/ 

p«\fect title 2nd the spooring agenr/ association. 

Instructional Program Title 

ImJtsA Co 5f7J CJfl ft* * prr A ts?# 

VocaCcnal Prc^am Area 






1. Level cf training, (check one c«3v) 

j~~) Secondly (regular m-schoo. program) 

H Pjst*Secondaiy (svc. ' i e for full-time training) 
j~| Adult (available icr part-time training) 
pi Mixed ! evel; *-cmbinaiicn of the above) Specify 



il Type of program: 

j~~j Regular vocational program 
f~| Spe.ial needs program 

Chech one Gr more items below to indicate per- 
sons served by this special program. 

a. Q Handicapped: 

Jj| Physical 
P| Mentally retarded 
fj Emotional 

f j Other (specify) _ 

b. FI Disadvantaged: 

j~j Socio-Economic 
P| Cultural 
Pj Educational 

P| 3*her (specif) __ . ... - 

3. Is this an MDTA training program? p] Yes pj No 

4. This instnjctional program is designed to train per- 
sons living in a: (check one) 

Pj Urban setting 

|~1 Rural setting 

Pj Both urban and rural settings 



i 5. Is this instructional program composed of a series of 
separate specialized courses? pj Yes jjj No 

C. How many courses must a student take to complete 
this in sim -fional program? 

P|~~ J Courses 

7. I it the individual vocational ^curses that compose 
tnis instroctiona. program. II these courses are 

sequenced sist in order. _ 



8. VJher* is this instructiviia. program offered? 

Pj Dayon.y 
pj Night only 
Pj Both day and night 

S. Dees this instructional pregram operate at times 
other than during trie regular school year (Sepi- 
June)? QYes Q No 

If yes, which months: 



10. What is the duration of time, by school years and 
months of instruction, required for a student to com- 
plete this instructional program? 

Pj Number of school years 
| p| Number of actual months of instruction 



